Skip to content

feat(storage)!: add live root-disk growth - #1533

Merged
appcypher merged 11 commits into
appcypher/checkpoint-restore-clonefrom
appcypher/live-root-disk-growth
Sep 10, 2026
Merged

feat(storage)!: add live root-disk growth#1533
appcypher merged 11 commits into
appcypher/checkpoint-restore-clonefrom
appcypher/live-root-disk-growth

Conversation

@appcypher

@appcypher appcypher commented Sep 6, 2026

Copy link
Copy Markdown
Member

TL;DR

Grow owned managed and flat ext4 root disks without restarting, including checkpoint-backed qcow2 roots. This is #7, stacked directly on #1503 with no newer main changes, and depends on libkrun #124.

Description

  • Reuse the existing modify API and preserve explicit restart and next-start policies; shrinking and cloud/user-owned disk restrictions are unchanged.
  • Briefly pause to extend the writable block head, then resume and expand the mounted filesystem through the guest agent.
  • Record pending growth for forward recovery and keep snapshotting and compaction blocked until it completes.
  • Support stopped growth using a private staging head without modifying sealed ancestors or adding chain depth; old private bindings currently remain on disk.
  • Preserve each layer's actual capacity so older snapshots restore at their captured size after later growth.
  • Add protocol generation 9 requests, user docs, a live matrix and measured phase results; pin libkrun commit ec9f119dcc144f1c011fe62ebd224d660c2191a3 for reproducible builds.
  • Remove redundant Rust modification setters without deprecation: use memory, max_memory and root_disk_size instead of the _mib setters or oci_upper_size* aliases. Bare MiB integers and typed sizes remain supported; other SDKs and serialized fields are unchanged. This is an intentional source-breaking API cleanup.
msb modify worker --root-disk 8G
msb modify worker --root-disk 16G --next-start
sb.modify().root_disk_size(8192).apply().await?;
await sb.modify(root_disk_size=8192)
await sandbox.modify({ rootDiskSize: 8192 });
_, err := sb.Modify(ctx, m.ModifyOptions{RootDiskSizeMiB: 8192})

Test Plan

  • cargo fmt --all -- --check and pinned offline CLI check pass.
  • cargo test --offline -p microsandbox --lib -- --test-threads=1: 667 passed, three ignored; cargo test --offline -p microsandbox-cli --lib commands::modify: ten passed. An unchanged stale-socket test failed in an earlier parallel suite and passed alone and serially.
  • Focused ext4, runtime disk and protocol schema tests pass, including four external e2fsck checks and 135 companion device tests.
  • Focused Clippy passes with only the pre-existing derivable_impls and too_many_arguments lint classes allowed.
  • macOS debug/release live matrices and 16 one-shot 512 MiB to 4 GiB/8320 MiB scenarios pass, including allocated data, full restore, cold boot, immutable ancestors and 16 independent filesystem checks. Live first-growth CLI calls took 74–86 ms; separate smaller-grow phase measurements had 3.8–8.1 ms VM pauses. These live runs preceded the setter-only cleanup. See scripts/smoke/reports/root-disk-growth-2026-09-06.md for coverage and limitations.
  • Linux/KVM and Windows/WHP live qualification, native language SDK runs and additional fault/load testing (source transfers await explicit destination approval).

Confidence Score: 5/5

The PR appears safe to merge because no blocking failure remains within the eligible follow-up-review scope.

No blocking failure remains.

Reviews (8): Last reviewed commit: "chore(stack): inherit Python snapshot li..." | Re-trigger Greptile

Extend the existing modify path to grow managed and flat ext4 roots,
including checkpoint-backed qcow2 heads, without changing sealed layers.
Persist pending growth for forward recovery and support staged stopped
growth over the same logical disk chain.

Add guest protocol generation 9 preflight and resize acknowledgments,
per-layer snapshot capacities, user documentation, and a macOS live
qualification harness and report. Pin the exact libkrun companion commit
until the capacity API is released.

Keep this change stacked directly on checkpoint-restore-clone (#6).
Linux and Windows qualification and source publication remain pending.
Exercise 512 MiB to 4 GiB and 8320 MiB across managed and flat roots,
raw and qcow2 heads, and live and stopped modification paths. Verify
allocated data, snapshot capacities, immutable ancestors and cold boots.

Add independent qemu-img and read-only e2fsck qualification on disposable
copies and record the macOS release timings and remaining coverage gaps.
No runtime behavior changes are included.
Use memory, max_memory and root_disk_size for both bare MiB integers
and typed sizes. Update CLI callers and documentation and verify both
input forms produce the same serialized patch values.

BREAKING CHANGE: remove memory_mib, max_memory_mib, root_disk_size_mib,
oci_upper_size and oci_upper_size_mib from SandboxModificationBuilder.
Use the unsuffixed setters; no deprecation aliases are retained.
Other SDK interfaces and wire fields are unchanged.
Rename snapshot create --from to --from-sandbox to match the existing
SDK source-sandbox naming. Update documentation and smoke scripts,
including the root-disk growth coverage.

Test the required source, full capture, direct archives, and rejection
of the old flag. Snapshot parser tests pass (7 tests).

BREAKING CHANGE: msb snapshot create requires --from-sandbox instead
of --from. No compatibility alias is provided.
Merge the corrected #6 parent, retain live root growth through the client/runner control split, and use the published 0.1.34 runtime dependencies. Preserve release migrations and generation-8 bulk framing while placing unreleased root-growth messages in generation 9. Keep current tuning documentation and restore the live-growth examples and recovery guarantees.
Pin msb_krun and msb_krun_utils to the published 0.1.35 release and refresh only their matching lockfile entries.

Pick up runtime resolution of optional macOS GIC APIs without changing the snapshot format, guest protocol, or stack ancestry.
Keep stdio guard construction in the I/O error domain so the runtime spawn helper compiles on Windows. Own successful flag changes immediately and roll them back if a later handle update fails.

Add Windows regression tests for partial failure, duplicate handles and unchanged non-inheritable handles, and run them in both Windows CI jobs.
Carry the selected release baseline, stack CI triggers and focused CI corrections through the existing full-checkpoint parent. Preserve live disk growth and Windows stdio cleanup behavior.
Carry full checkpoint recovery and portability fixes into root growth.
Reconcile only the invalid-head tests: valid checkpoint chains remain
growable and malformed chains must leave base, head and journal unchanged.
Include the final full-checkpoint parent and its inherited CLI and
manifest cleanup without changing root-disk growth behavior.
Carry the verified parent-only Python binding cleanup into root-disk growth without changing its public APIs or runtime behavior.
@appcypher
appcypher merged commit 1305756 into releases/v0.7.0 Sep 10, 2026
80 checks passed
@appcypher
appcypher deleted the appcypher/live-root-disk-growth branch September 10, 2026 19:42
appcypher added a commit that referenced this pull request Sep 10, 2026
## TL;DR

Add direct local branching, explicit CoW full-snapshot restore, resident
pause/resume, and preferred `.msb` archive naming as stack item #8. The
September 9 platform report records fixes for the earlier Windows ARM64
restore hang and private-memory gaps. Final integrated-head
qualification is tracked separately below; Windows x86-64 remains
untested.

## Description

- Stack directly on #1533 (`appcypher/live-root-disk-growth`) without
importing unrelated Microsandbox main changes.
- Add `msb branch SOURCE --name CHILD` and matching SDK methods for
independent local children without publishing a durable full snapshot. A
running source resumes after capture; a user-paused source stays paused.
- Add restore-only `--forked` for private CoW memory on supported Linux,
macOS, and Windows hosts, backed by a protected immutable cache with
retained file ownership and reflink reuse where available. No source
creation-time memory option is required; the superseded unreleased
memory-snapshot API is removed.
- Add pause/resume through the existing host control endpoint across
CLI, Rust, Python, TypeScript, and Go, including repeated full capture
while retaining a user pause.
- Preserve ordinary resume identity, correct guest wall clock before
workload thaw, and retain recovery ownership after uncertain freeze/thaw
failures.
- Prefer `.msb` in help and documentation while retaining content-based
archive detection, legacy suffixes, explicit filenames, and plain-tar
output.
- Use published registry dependencies: `msb_krun` / `msb_krun_utils`
0.1.35, `msb-vm-memory` 0.18.0-msb.2, and `msb-imago` 0.1.7. Remove the
development Cargo Git patches; retain the matching libkrunfw submodule
pin.
- Retain the historical September 9 platform matrix in
`scripts/smoke/reports/cow-platform-fixes-2026-09-09.md`: eager/forked
restore and direct branching pass on macOS ARM64, Linux x86-64, Linux
ARM64/nested KVM, and Windows ARM64. These results apply to their
recorded revisions, not a fresh full-platform qualification of 0.1.35 or
subsequent integration changes.
- Expose standalone running/paused disk-only capture through `msb
snapshot create NAME --from-sandbox SOURCE`, including direct archive
capture, without capturing RAM. It is crash-consistent; see
`scripts/smoke/reports/live-disk-snapshot-2026-09-09.md` for the
three-host flat/layered checks.
- Store snapshots as immutable group members: the source sandbox name is
the default group, `--group` selects another, and `group:member` selects
an exact checkpoint. Bare groups resolve their selected heads;
multi-archive imports resolve dependencies within the batch and
explicitly selected group.
- Support base-relative export with `--since` (disk layers and reused
RAM objects), disk-layer selection with `--last-layers`, explicit-base
restore, and manual live/stopped disk compaction. Successive disk-only
capture/export still has the documented layer-ID limitation;
full-checkpoint incremental exports are unaffected.

## CLI examples

### Branch a running sandbox

```bash
msb create alpine --name source --memory 512M
msb exec source -- sh -c 'echo original > /dev/shm/message'

# Capture current execution and start an independent child.
msb branch source --name child
msb exec child -- cat /dev/shm/message
# original

# Child writes stay private.
msb exec child -- sh -c 'echo changed > /dev/shm/message'
msb exec source -- cat /dev/shm/message
# original

# A child can itself be branched, including its private changes.
msb branch child --name grandchild
msb exec grandchild -- cat /dev/shm/message
# changed
```

`branch` captures RAM, disk, CPU and device state with a brief source
pause, then starts a detached child. CoW is inherent: no `--forked`
option is needed. It writes local backing but does not create a
registered, durable full snapshot. Branching requires a new child name
and currently rejects sources with published host ports.

### Branch a paused source

```bash
msb pause source
msb branch source --name paused-child
# The child runs; source remains paused.
msb resume source
```

`pause` and `resume` keep the VM resident; neither creates a snapshot.

### Save a durable full snapshot, then restore children

```bash
msb snapshot create saved --from-sandbox source --full

# Resume captured execution using private CoW memory.
msb create --name restored-forked --from-snapshot source:saved --forked

# Resume the same captured execution with eagerly copied memory.
msb create --name restored-eager --from-snapshot source:saved
```

The snapshot is installed in group `source`, so its exact selector is
`source:saved`.

`--forked` selects the restored child's memory backing, not the
snapshot's contents. It requires a full snapshot and cannot be combined
with a fresh image boot or `--disk-only`.

### Capture and restore directly through an archive

```bash
# Save directly to .msb without installing a snapshot directory.
msb snapshot create portable --from-sandbox source --full --archive ./portable.msb

# Restore directly into child staging, without first loading a snapshot directory.
msb create --name archive-child --from-snapshot ./portable.msb --forked

# Alternatively, use only the captured disk and perform a fresh boot.
msb create --name disk-child --from-snapshot ./portable.msb --disk-only
```

### Export changes and compact explicitly

```bash
msb snapshot create later --from-sandbox source --full
msb snapshot save source:later changes.msb --since source:saved
msb snapshot load changes.msb --base source:saved --group received
msb create --name imported-child --from-snapshot received:later --forked

# Inspect and apply compaction; the count includes the base, never the writable head.
msb modify source --compact --layers 3 --dry-run
msb modify source --compact --layers 3
```

After compaction, export a new standalone baseline before exporting
further deltas. This does not invalidate previously captured snapshots.

## SDK examples

Each language also exposes branching on sandbox handles returned by
`get`. The following examples assume an existing sandbox named `source`;
full-restore examples use `source:saved`, the grouped snapshot created
above.

Rust, inside an async fallible context:

```rust
use microsandbox::Sandbox;

let source = Sandbox::get("source").await?;
let child = source.branch("rust-child").await?;
let grandchild = child.branch("rust-grandchild").await?;
let restored = Sandbox::builder("rust-restored")
    .from_snapshot("source:saved")
    .forked()
    .create().await?;
```

Python, inside an async function:

```python
from microsandbox import Sandbox

source = await Sandbox.get("source")
child = await source.branch("python-child")
grandchild = await child.branch("python-grandchild")
restored = await Sandbox.create(
    "python-restored", from_snapshot="source:saved", forked=True
)
```

TypeScript:

```ts
import { Sandbox } from "microsandbox";

const source = await Sandbox.get("source");
const child = await source.branch("node-child");
const grandchild = await child.branch("node-grandchild");
const restored = await Sandbox.builder("node-restored")
  .fromSnapshot("source:saved").forked().create();
```

Go, using `m` for `github.com/superradcompany/microsandbox/sdk/go` and
an existing context:

```go
source, err := m.GetSandbox(ctx, "source")
if err != nil { return err }
child, err := source.Branch(ctx, "go-child")
if err != nil { return err }
defer child.Close()
restored, err := m.CreateSandbox(ctx, "go-restored",
    m.WithFromSnapshot("source:saved"), m.WithForked())
if err != nil { return err }
defer restored.Close()
```

Stop the created children explicitly when finished; branching creates
detached children.

## Test Plan — historical evidence

The checked items below record prior runs at their documented revisions.
They are not a claim that the final integrated branch with libkrun
0.1.35 and the latest transport changes has completed every test.

- [x] Registry-only dependency update: CLI build, 32 checkpoint tests,
39 snapshot tests, formatting, and the 56-step Mac
direct-branch/full-restore/growth/compaction run. See
`scripts/smoke/reports/registry-dependencies-2026-09-09.md`;
Linux/Windows live matrices were not rerun for this packaging update.
- [x] Prior macOS/HVF release-build direct-branch checks: RAM/disk
isolation, branch-of-branch after private writes, running/paused
sources, backing ownership and same-name races; Python/TypeScript/Go
native lifecycle smoke tests. Harnesses:
`scripts/smoke/cli/direct-branch.py` and
`scripts/smoke/cli/branch-ownership.py`.
- [x] September 9 direct-branch and `--forked` CLI matrices on Linux
x86-64, Linux ARM64/nested KVM, and Windows ARM64; backing lifetime,
failed-restore refusal, sibling isolation, and timer-progress
regressions. See `cow-platform-fixes-2026-09-09.md` for exact coverage
and limitations.
- [x] `cargo fmt --all -- --check` and `git diff --check`.
- [x] `PYTHONPYCACHEPREFIX=/private/tmp/msb-stack8-pycache python3 -m
py_compile scripts/smoke/cli/cow-memory-lifecycle.py`.
- [x] `cargo test -p microsandbox-runtime --lib --offline` (179 tests;
rerun outside the filesystem sandbox for temporary IPC sockets).
- [x] `cd sdk/node-ts && npm run test:unit && npm run typecheck` (137
tests).
- [x] Prior live CLI matrices on Linux/KVM and macOS/HVF, macOS
Python/Node/Go SDK smoke tests, and Linux 256→384→256→512→256 MiB CoW
resize with subsequent full lifecycle checks, as recorded in the report.
- [x] Windows ARM64 eager and forked full restore remain usable for
guest commands in the corrected September 9 matrix. The old failure
description and any older automated-review wording below are superseded
by that report.

## Current transport and integration qualification

Pushed head `5f46d63e` contains runtime code tested at `eada7e20`; the
only subsequent change expands the Windows CI filter to run both
lifecycle-handoff and stdio-guard regressions. It includes the updated
#7 base, libkrun 0.1.35 and the transport fixes. Earlier live results
below are revision-scoped.

- [x] Integrated checks: 285 runtime, 66 protocol, and 31 migration
tests passed, plus Clippy. Release build passed in 3m24s.
- [x] Frozen transport-fix revision `f014`: macOS and three Linux
transport cohorts passed their normal and combined live matrices. These
are not final-head/libkrun 0.1.35 qualification.
- [x] All 16 added `SnapshotArchive` invalid-receiver/constructor probes
passed against an existing native binary. Initial integrated CodeQL
gates were neutral/successful; the baseline alert was not suppressed.
Fresh analysis and native CI remain separate checks.
- [x] Exact integrated-code Mac release run: nine of nine pipe/PTY/TCP
pause, full-restore and branch cases passed in 200.552 seconds including
setup and cleanup. Fresh commands completed before autonomous input
gates. No owned runtime processes or resident sandboxes remained;
temporary homes were removed.
- [ ] Complete fresh stacked CI and final Linux/Windows ARM64
qualification. Latest Windows changes have not been live-rerun.
- [ ] Confirm registry-only packaged/installed smoke with matching
artifacts and no development overrides.

Integrated Mac SHA256: runtime
`f89c4ec3bf154b7daaf077c8185d98ffe37db02d6843635cc4c54e155a6cc339`;
agentd
`3219165916e536620519b8a0a543c53a9449ba07635016153a8193444f2a41cf`;
firmware
`ea0d458cdc12a0fa6dac8d192542ddc39717f816da41176582905e31a8bf868c`.

Performance is mixed: frozen Linux all-load PTY median improved 19.72%,
but loaded-ping median increased from 0.066 to 1.649 ms during one
upload trial, and small TCP latency also increased. These remain
performance follow-ups, not a universal speedup claim.

Broader outstanding qualification remains Windows x86-64, NUMA
combinations, physical-memory sharing/reclamation under pressure,
broader concurrency/failure injection, and repeated release-build
performance measurements.

<!-- greptile_comment -->

<details><summary><h3>Confidence Score: 5/5</h3></summary>

The PR appears safe to merge because no blocking failure remains in the
eligible follow-up review scope.

No blocking failure remains.
</details>

<sub>Reviews (6): Last reviewed commit: ["build(deps): use published
snapshot
runt..."](75429fe)
| [Re-trigger
Greptile](https://app.greptile.com/api/retrigger?id=61309270)</sub>

<!-- /greptile_comment -->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant